Information System and I/O Processing Method

ABSTRACT

First and second storage apparatuses respectively and internally set a remote copy pair for copying data of a first volume to a second volume and associate a third volume with a remote copy pair according to an external command. When an I/O request to the first volume ends in an error, a host computer sends an I/O request directed to the second volume to the second storage apparatus. If the first and the second storage apparatuses detect a failure in the first or the second storage apparatus of the other side or a connection failure between the first and the second storage apparatuses, the first and the second storage apparatuses store in the third volume a failure information flag showing that the failure was detected.

CROSS-REFERENCES

This application relates to and claims priority from Japanese PatentApplication No. 2008-117646, filed on Apr. 28, 2008, the entiredisclosure of which is incorporated herein by reference.

BACKGROUND

The present invention relates to an information system including aplurality of storage systems.

In general, storage apparatuses using HDD (hard disk drives) as thestorage device are mounted in an information system, and storage systemsincluding such storage apparatuses are accessed from a plurality of hostsystems (for instance, hosts) via a storage area network (SAN).Generally speaking, storage apparatuses adopt a method of attaining highreliability according to RAID (Redundant Array of Independent (orInexpensive) Disks) technology in order to provide the storageapparatuses with reliability that exceeds the reliability of stand-aloneHDDs. Nevertheless, pursuant to the advancement of information societyin recent years, the availability (service continuity) of informationsystems serviced with the reliability provided by the foregoing RAIDtechnology is becoming inadequate.

As technology for realizing high availability to deal with the foregoingsituation, the specification of U.S. Pat. No. 7,058,731 describestechnology where a host and two storage systems are respectivelyconnected, and data is remotely copied from one storage system to theother storage system. The entire disclosure of such reference isincorporated herein by reference.

SUMMARY

With the foregoing technology, if one of the storage systems fails andstops, it may be switched to another storage system to continuebusiness. Nevertheless, in this case, the host requires specialalternate path software for switching the storage systems, and this isnot a versatile approach.

Thus, an object of the present invention is to improve the availabilityof an information system including a storage system for performingremote copy between two or more storage systems, and a host which doesnot include any special alternate path software.

In order to achieve the foregoing object, the present invention providesan information system comprising a host computer, first storageapparatus connected to the host computer and including a first volume, asecond storage apparatus connected to the host computer and the firststorage apparatus and including a second volume, and a third storageapparatus connected to the first storage apparatus and the secondstorage apparatus and including a third volume. The first and the secondstorage apparatuses respectively, internally set a remote copy pair forcopying data of the first volume to the second volume, and associate thethird volume with the remote copy pair according to an external command.When an I/O request to the first volume ends in an error, the hostcomputer sends an I/O request directed to the second volume to thesecond storage apparatus. If the first and the second storageapparatuses detect a failure in the first or the second storageapparatus of the other side or a connection failure between the firstand the second storage apparatuses, the first and the second storageapparatuses store in the third volume a failure information flag showingthat a failure was detected.

The present invention additionally provides an I/O processing method inan information system comprising a host computer, a first storageapparatus connected to the host computer and including a first volume, asecond storage apparatus connected to the host computer and the firststorage apparatus and including a second volume, and a third storageapparatus connected to the first storage apparatus and the secondstorage apparatus and including a third volume. The I/O processingmethod comprises a first step of the first and the second storageapparatuses respectively, internally setting a remote copy pair forcopying data of the first volume to the second volume, and associatingthe third volume with the remote copy pair according to an externalcommand, a second step of the host computer sending, when an I/O requestto the first volume ends in an error, an I/O request directed to thesecond volume to the second storage apparatus, and a third step of thefirst and the second storage apparatus storing in the third volume, if afailure in the first or the second storage apparatus of the other sideor a connection failure between the first and the second storageapparatuses is detected, a failure information flag showing that thefailure was detected.

Accordingly, since an I/O request is sent to the second volume when anI/O request to the first volume ends in an error, it is possible toeffectively prevent the host computer from executing processing to anI/O request based on data of the second volume, which is not up to date,as a result of the remote copy with the first volume not being performedand data of the second volume not being up to date.

According to the present invention, it is possible to improve theavailability of an information system.

DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing an example of the hardwareconfiguration of an information system according to an exampleembodiment of the present invention;

FIG. 2 is a conceptual diagram showing an example outline of an exampleembodiment of the present invention;

FIG. 3 is a block diagram representing an example software configurationin a virtual storage apparatus, storage apparatus and failure detectionstorage apparatus;

FIG. 4 is a conceptual diagram representing an example pair status of anexample remote copy and an example transition of the pair status;

FIG. 5 is a flowchart showing an example I/O request processing; and

FIG. 6 is a flowchart showing an example I/O request processing.

DETAILED DESCRIPTION

Embodiments of the present invention are now explained in detail withreference to the attached drawings.

(1) First Example Embodiment 1. Configuration of Information System

FIG. 1 is a diagram showing an example of the hardware configuration ofan information system 100 according to an example embodiment of thepresent invention.

The information system 100 is configured from a storage apparatus 1500,a host computer (hereinafter abbreviated as “host”) 1100, a managementhost 1200, two or more virtual storage apparatuses 1000, and so on. Thenumber of storage apparatuses 1500, hosts 1100, and management hosts1200 may be one or more, respectively. The virtual storage apparatus1000 and the host 1100 are mutually connected via an I/O network 1300.The virtual storage apparatus 1000 and the storage apparatus 1500 andthe management host 1200 are mutually connected via a management network(not shown) or the I/O network 1300.

The host 1100 has a host internal network 1104, and connected to thisnetwork 1104 are a processor (abbreviated as “Proc” in FIG. 1) 1101, amemory (abbreviated as “Mem” in FIG. 1) 1102, and an I/O port(abbreviated as “I/O P” in FIG. 1) 1103. The management host 1200 mayalso adopt the same hardware configuration as the host 1100. Anexpansion card for adding an I/O port to the host 1100 is sometimesreferred to as an HBA (Host Bus Adapter).

The management host 1200 has a display device (not shown), and thisdisplay device is able to display a screen for managing the virtualstorage apparatus 1000 and the storage apparatus 1500. The managementhost 1200 is also able to receive a management operation request from auser (for instance, the operator of the management host 1200), and sendthe received management operation request to the virtual storageapparatus 1000 and the storage apparatus 1500. The management operationrequest is a request for operating the virtual storage apparatus 1000and the storage apparatus 1500 and there are, for example, operationsrelating to a parity group creation request, an internal LU (LogicalUnit) creation request, a path definition request, and a virtualizationfunction.

Although a fibre channel is foremost considered for the connection ofthe I/O network 1300, in addition, a combination of FICON (FibreCONnection: registered trademark) and Ethernet (registered trademark)with TCP/IP (Transmission Control Protocol/Internet Protocol) and iSCSI(internet SCSI (Small Computer System Interface)), or a combination ofEthernet (registered trademark) and network file systems such as NFS(Network File System) and CIFS (Common Internet File System) may also beconsidered. Moreover, the I/O network 1300 may be a differentcommunication device so as long as it is able to transfer the I/Orequest between network entities. The network connecting the virtualstorage apparatus 1000 and the storage apparatus 1500 may be the same asthe I/O network 1300.

The virtual storage apparatus 1000 includes a controller (indicated as“CTL” in FIG. 1) 1010, a cache memory (indicated as “CM” in FIG. 1)1020, and a plurality of HDDs 1030. As a preferred mode, the controller1010 and the cache memory 1020 are respectively configured from aplurality of components. This is so that, even if a failure occurs in astand-alone component and such component is blocked, I/O requests asrepresented by the reading and writing of data can be continuouslyreceived using the remaining components.

The controller 1010 is an apparatus (for instance a circuit board) forcontrolling the operation of the virtual storage apparatus 1000. Thecontroller 1010 has an internal network 1017, and connected to thisinternal network 1017 are an I/O port 1013, a cache port (indicated as“CP” in FIG. 1) 1015, a management port (indicated as “MP” in FIG. 1)1016, a backend port (indicated as “B/E P” in FIG. 1) 1014, a processor(for instance a CPU (Central Processing Unit)) 1011, and a memory 1012.The controllers 1010 and the cache memory 1020 are mutually connectedvia the storage internal network 1050. The controller 1010 and each HDD1030 are mutually connected via a plurality of backend networks 1040.

The hardware configuration of the storage apparatus 1500 may beconfigured from the same type of components as the virtual storageapparatus 1000. If the virtual storage apparatus 1000 is a dedicatedvirtualization apparatus or a switch that is not equipped with an HDD,the storage apparatus 1500 does not need to be configured from the sametype of components as the virtual storage apparatus 1000. The internalnetwork of the host 1100 and the virtual storage apparatus 1000 ispreferably a broader band than the transfer band of the I/O port 1013,and all or a part thereof may be substituted with a bus or a switch-typenetwork. Although FIG. 1 shows one I/O port 1013 in the controller 1010,in reality a plurality of I/O ports 1013 may exist in the controller1010.

Based on the foregoing hardware configuration, the host 1100 will beable to read and write all or a part of the data retained in the virtualstorage apparatus 1000 or the HDD of the storage apparatus 1500. In theensuing explanation, the system in charge of data storage is referred toas a storage cluster. In addition, a subsystem that realizes highavailability by including two types of systems in the storage clusterand which includes one or both of the virtual storage apparatus 1000 andthe storage apparatus 1500, is referred to as a storage subsystem.

2. Outline of Present Example Embodiment

In this example embodiment, in order to improve the availability of astorage system including the virtual storage apparatus 1000 with avirtualization function for virtualizing a storage area of a volume orthe like in another storage apparatus 1500, a duplexed configurationusing another virtual storage apparatus 1000 is adopted. FIG. 2 is adiagram showing an outline of this duplexed configuration.

In this outline, the storage system includes a virtual storage apparatus1000L, a virtual storage apparatus 1000R, a storage apparatus 1500L, anda storage apparatus 1500R. For ease of understanding in the ensuingexplanation, let it be assumed that the virtual storage apparatus 1000Land the storage apparatus 1500L function as the primary system(production system), and the virtual storage apparatus 1000R and thestorage apparatus 150OR function as the secondary system (backupsystem). Nevertheless, if there are two or more volumes to be providedby the respective virtual storage apparatuses 1000L, 1000R to the host1100, in substitute for the primary system and the secondary systembeing handled in virtual storage apparatus units, it would suffice so aslong as the virtual storage apparatuses 1000L, 1000R to handle theprimary system in volume units are defined.

The respective virtual storage apparatuses 1000L, 1000R provide a partor all of the areas of a parity group (configured with RAID technology)that uses its own HDD 1030 as the system component as a volume 3000LA,3000RA to the host 1100 (corresponds to the portion indicated as ‘A’ inthe cylinder of FIG. 2). The virtual storage apparatus 1000 may alsooptionally provide a virtual volume 3000LB, 3000RB (volume in which anonvolatile storage area of a corresponding HDD or the like existsoutside the virtual storage apparatuses 1000L, 1000R) based on thevirtualization function (corresponds to the portion indicated as ‘B’ inthe cylinder of FIG. 2). In this outline, a part or all of the volumes3500LB, 3500RB provided by the storage apparatuses 1500L, 150OR are usedas the corresponding nonvolatile storage area.

In the ensuing explanation, the term “volume data” includes, in additionto the data stored in the HDD 1030, data that is temporarily stored inthe cache memory 1020. Moreover, the term “virtual volume data”described later includes, in addition to the data stored in the volumes3500LB, 3500RB of the storage apparatuses 1500L, 1500R, data that istemporarily stored in the cache memory 120 of the virtual storageapparatuses 1000L, 1000R.

Meanwhile, an application program (hereinafter sometimes abbreviated as“application”) 2010 and an OS (Operating System) 2020 are running on thehost 1100. If there are a plurality of access paths in a single volume3000, alternate path software with a function of collectively handlingsuch plurality of access paths may be running as a part of the OS 2020.

The application 2010 uses an interface provided by the OS 2020 andaccesses data in the virtual storage apparatus 1000. The OS 2020interprets the access request from the application 2010 and, in order toidentify the requested data, designates a volume address (for instance,represented as a combination of a target port address and LUN with aSCSI standard) of the volume 3000 provided by the virtual storageapparatus 1000 and a data location (shown as LBA with a SCSI standard)in the volume 3000, and issues an I/O request to the virtual storageapparatus 1000.

The virtual storage apparatus 1000 accesses (reads or writes) the datastored in the designated LBA of the designated volume 3000, and replieswith the result to the host 1100. The OS 2020 of the host 1100 receivesthe result returned from the virtual storage apparatus 1000, and returnsthe result to the application 2010.

The alternate path software intervenes in the exchange between theapplication 2010 and the OS 2020, and has a function of selecting theaccess path to be used when a single volume comprises a plurality ofaccess paths. With the SCSI standard, if one volume 3000 comprises aplurality of access paths, a different volume address (that is, acombination of the target port address and LUN) is allocated to therespective access paths. In other words, a single volume will have aplurality of volume addresses.

The alternate path software selects one of the volume addresses (thatis, one of the access paths) and requests the OS 2020 to issue an I/Orequest. The OS 2020 uses the access path selected by the alternate pathsoftware, issues an I/O request to the virtual storage apparatus 1000,receives the result from the virtual storage apparatus 1000, and returnsthe result to the alternate path software. The alternate path softwarereturns the result as is to the application 2010 if the result is an I/Osuccess. If the result is an I/O error, the alternate path softwareselects a separate access path and once again requests the OS 2020 toissue an I/O request. If all access paths are subject to I/O error, theI/O error is replied to the application 2010.

The alternate path software uses a volume identifier as an identifierthat is unique to the volume (hereinafter referred to as a “uniquevolume identifier”) in order to know whether a plurality of volumeaddresses are associated with a single volume. The alternate pathsoftware makes an inquiry to the virtual storage apparatus 1000regarding the volume identifier of each of the plurality of volumeaddresses. With the SCSI standard, an Inquiry command is used. Thevirtual storage apparatus 1000 replies with the unique volume identifierto the alternate path software. Since the volume identifier is a uniquevolume identifier, even if the volume address is different, the sameidentifier is returned if the volume 3000 is the same. Thereby, thealternate path software is able to recognize that the volume addressesreturning the same volume identifier are the plurality of access pathsto a single volume.

The alternate path software also acquires the priority of the accesspaths. With the SCSI standard, the priority is returned to the alternatepath software together with the identifier as the reply of the Inquirycommand. Generally, priority is represented in the two stages of high orlow. In other words, the alternate path software is able to obtaininformation such as a certain access path having high priority and aseparate access path having low priority. When the alternate pathsoftware acquires the priority, it selects an access path with highpriority upon selecting the access path, and requests the OS 2020 to usethe selected access path and issue an I/O request. If there are three ormore access paths, the access path to be used is foremost selected amongthe access paths with high priority. The alternate path software selectsan access path with low priority and requests the OS 2020 to issue anI/O request only when all I/O requests using access paths with highpriority ends in an error.

The volumes 3000LA, 3000RA, 3000LB, 300ORB provided by the virtualstorage apparatuses 1000L, 1000R shown in FIG. 2 are of a remote copyrelationship. Remote copy is a function loaded in the virtual storageapparatus 1000 of copying write data received by the volume 3000 of acertain virtual storage apparatus 1000 to the volume 3000 of anothervirtual storage apparatus 1000. The copy source volume is referred to asa primary volume, and the copy destination volume is referred to as asecondary volume.

In this embodiment, the virtual storage apparatus 1000L is set as thevirtual storage apparatus 1000 with high priority, and the virtualstorage apparatus 1000R is set as the virtual storage apparatus 1000with low priority. In FIG. 2, the volumes 3000LA, 3000LB are primaryvolumes, and the volumes 3000RA, 300ORB are secondary volumes.

Although the basic function of the remote copy in this embodiment issomewhat equivalent to standard remote copy, certain functions differfrom the standard remote copy. Some important characteristic functionsof this embodiment are explained below. Foremost, with the remote copyof this example embodiment, the volume identifier of the primary volumeand the volume identifier of the secondary volume are made to be thesame. Specifically, the volume 3000LA and the volume 3000RA reply withthe same volume identifier to the SCSI Inquiry command. Consequently,the alternate path software of the host 1100 recognizes the volumes3000LA and 3000RA, which are different volumes under ordinarycircumstances, as a single volume, and recognizes the respective volumesas a plurality of access paths to that single volume.

Regarding the priority of access paths also, the primary volume replieswith an access path with high priority and the secondary volume replieswith an access path with low priority. As a result of adopting this kindof configuration, since the alternate path software will process the I/Ousing an access path with high priority, it will consequently issue anI/O request to the primary volume. The alternate path software selects alow priority access path only when the I/O with all high priority accesspaths ends in an error (that is, only when the I/O to the primary volumeends in an error); in other words, it issues an I/O request to thesecondary volume.

One characteristic function of the secondary volume in the remote copyof this embodiment is that, upon receiving an I/O request from the host1100, the remote copy is stopped and the I/O processing is started. If ahost I/O is issued to the secondary volume based on the foregoing volumeidentifier and the access path priority, this means that the I/O to theprimary volumes (that is, to the high priority access paths) all endedin an error. Here, by starting the I/O processing with the reception ofthe I/O request by the secondary volume as the trigger, the alternatepath software will determine that the I/O was successful as a result ofselecting a low priority access path. Thereby, a failover of the virtualstorage apparatus 1000 can be realized with the functions available instandard alternate path software.

Like this, with the information system 100, the alternate path softwareis not given a special function, and the switching of the virtualstorage apparatus 1000L and the virtual storage apparatus 1000R isrealized by the virtual storage apparatus 1000L and the virtual storageapparatus 1000R sending a reply to the I/O request in accordance withthe functions of existing alternate path software.

Although the processing subject of the various types of processing ofthe application 2010, the OS 2020, and the alternate path software wasexplained as a program in order to clarify the processing contents ofthe processor 1011 of the host 1100 based on the respective programs, inreality, it goes without saying that the processor 1011 of the host 110performs the foregoing processing based on the relevant programs.

3. Example Programs to be Executed with Virtual Storage Apparatus 1000and Example Information Managed by Programs

FIG. 3 is a diagram shows example programs to be executed by the virtualstorage apparatuses 1000 (1000L, 1000R) and the storage apparatuses 1500(1500L, 1500R), and example information to be managed by these programs.Although these programs are retained in the memory 1012 (FIG. 1), theprocessor 1011 (FIG. 1), and the cache memory 1020 and executed, theprograms may also be partially configured as hardware and executed.

3.1. Example I/O Processing Program 6020, 6120, Parity Group Information6060, 6160 and Volume Information 6050, 6150

The example parity group information 6060 includes information relatedto the following configuration for each parity group.

(1) Identifiers of the HDDs 1030 configuring the parity group. Since aplurality of HDDs 1030 are participating in the parity group, aplurality of identifiers (information) exist for each parity group.

(2) RAID level

The volume information 6050 includes information related to thefollowing configuration for each volume.

(1) Volume capacity

(2) Identifier of parity group retaining data corresponding to thevolume and area (start address and/or end address) in the parity group.

The I/O processing program 6020 refers to the volume information 6050and the parity group information 6060 and executes the followingprocessing regarding the I/O request received from the host 1100.

(A) Staging: Data stored in the HDD 1030 is copied in the cache memory1020.

(B) Destaging: Data stored in the cache memory 1020 is copied to the HDD1030. Redundant data may be created with RAID technology aspreprocessing.

(C) Read processing: In response to a read request received from thehost 1100, whether data corresponding to that request exists in thecache memory 1020 is determined. If data corresponding to that requestdoes not exist in the cache memory 1020, staging processing is executedto copy that data in the cache memory 1020, and that data is thereaftersent to the host 1100. If corresponding data exists in the cache memory1020, that data is sent to the host 1100.

(D) Write processing: Write data 6210 received from the host 1100 isstored in the cache memory 1020. If the unused area in the cache memory1020 is insufficient during the processing, destaging processing isexecuted to copy appropriate data to the HDD 1030, and that area isthereafter diverted to the cache memory 1020. If the area previouslystored in the cache memory 1020 is included in the write request, thereare cases where such area is overwritten on the area in the existingcache memory 1020.

(E) Cache algorithm: Data in the HDD 1030 to be staged and data in thecache memory 1020 to be destaged are determined according to analgorithm such as LRU based on the frequency or timing of referral ofdata in the cache memory 1020.

3.2. Example Virtualization Program 6030 and Virtualized Information6070

The virtualized information 6070 includes information related to thefollowing configuration for each virtual volume.

(1) Information on the area in the volume of the storage apparatus 1500and information concerning the area to be provided to the host 1100 aswhich area of the address space in the virtual volume. If theconfiguration includes a plurality of virtual volumes, the followinginformation also exists in a plurality.

(1-1) Identifier of the storage apparatus 1500 (or identifier of theport) configuring the virtual volume, identifier of the volume, and areain the volume (start address and end address)

(1-2) Areas (start address and end address) in virtual volume

(2) Capacity of virtual volume

The virtualization program 6030 is a program for the virtual storageapparatus 1000 to provide a volume to the host 1100 using the volumeprovided by the storage apparatus 1500. As the correspondencerelationship of the virtual volume 3000B(don't see this reference numberin drawings?) provided by the virtualization program 6030 and the volume3500B in the corresponding storage apparatus 1500, there are thefollowing patterns.

(A) A case of using the overall volume in the storage apparatus 1500 asthe storage area of the virtual volume 3000B(don't see this referencenumber in drawings; see many additional occurrences in spec). In thiscase, the capacity of the virtual volume will be approximately the samecapacity as the selected volume 3500B (Case of storing controlinformation and redundant information in the volume of the storageapparatus 1500. If there is no such information, this will be the samecapacity).****

(B) A case of using a partial area of the volume in the storageapparatus 1500 as the storage area corresponding to the virtual volume3000B. In this case, the capacity of the virtual volume 3000B will beroughly the same as the area capacity to be used.

(C) A case of combining a plurality of volumes 3500B in a plurality ofstorage apparatuses 1500 and using this as a storage area of the virtualvolume 3000B. In this case, the capacity of the virtual volume 3000Bwill be roughly the same as the total value of the respective volumecapacities. As the combination method, striping, Concatenate (connectinga plurality of volumes and handling them as a single volume) or the likemay be used.

(D) A case of storing parity information and mirror data incidental tothe pattern of (C). In this case, the capacity of the virtual volume3000B will be half of (C) when storing mirror data, and will depend onthe parity calculation method when storing parity. By combining the highreliability based on RAID in the storage apparatus 1500, reliabilityregarding the data stored in the virtual volume 3000B can be improvedeven further.

In all of the foregoing patterns, the storage apparatus identifier (orport identifier) and the volume identifier (information used in the I/Orequest for identifying the volume in the virtual storage apparatus orunder the control of the port; LUN (Logical Unit Number), and the CUnumber in CKD format and the LDEV (Logical DEVice) number, etc.) willdiffer from the original volume.

The virtualization program 6030 is called by the I/O processing program6020 when the data to be subject to staging or destaging corresponds tothe virtual volume, and executes the following processing using thevirtualized information 6070.

(A) Staging: After deciding from which storage apparatus 1500 the datastored in its volume 3500B is to be copied in the cache memory 1020based on the correspondence relationship of the virtual volume 3000B andthe volume 3500B of the storage apparatus 1500, data is copied in thecache memory 1020.

(B) Destaging: After deciding to which volume 3500B of which storageapparatus 1500 the data stored in cache memory 1020 is to be copiedbased on the correspondence relationship of the virtual volume 3000B andthe volume 3500B of the storage apparatus 1500, data is copied to thevolume 3500B of the storage apparatus 1500. Here, redundant data may becreated with RAID technology as preprocessing.

3.3. Remote Copy Program 6010 and Copy Pair Information 6040

The copy pair information 6040 retains the following information foreach copy pair (sometimes abbreviated as “pair”) of the primary volumeand the secondary volume of remote copy. In this embodiment, targetvolumes for realizing high availability are designated as the primaryvolume and the secondary volume.

(1) Identifier of the virtual storage apparatus 1000 retaining theprimary volume and identifier of the volume

(2) Identifier of the virtual storage apparatus 1000 retaining thesecondary volume and identifier of the volume

(3) Copy pair status (to be described in detail later)

The remote copy program 6010 is a program for mirroring the data storedin the primary volume to the secondary volume, and performs processingby referring to the copy pair information 6040. The outline ofprocessing and the pair status of remote copy (in particular synchronousremote copy) are explained below.

3.3.1. Copy Processing Operation of Synchronous Remote Copy

Synchronous remote copy is a method of remote copy where, as describedabove, when the copy source virtual storage apparatus 1000 receives awrite request from the host 1100 for writing data into the primaryvolume, it sends the write data to the copy destination virtual storageapparatus 1000, and thereafter returns a write request completion to thehost 1100.

When synchronous remote copy is to be executed, the controller 1010 ofthe virtual storage apparatus 1000 manages information referred to as acopy pair status (Simplex, Initial-Copying, Duplex, Suspend andDuplex-Pending) in order to display the remote copy status between thepair of primary volume and secondary volume on the management host 1200,or operate the remote copy status. FIG. 4 shows a status transitiondiagram concerning the pair status of synchronous remote copy. Each pairstatus is explained below.

3.3.1.1. Simplex Status

The Simplex status is a status where the copy between the primary volumeand secondary volume configuring a pair has not started.

3.3.1.2. Duplex Status

The Duplex status is a status where the synchronous remote copy has beenstarted, the initialization copy described later is complete, and thecontents of the primary volume and the contents of the secondary volumeconfiguring a pair are identical. In this status, excluding the areathat is currently being written, the contents of the primary volume dataand the contents of the secondary volume data will be the same. DuringDuplex and in the Duplex-Pending status and the Initial-Copying status,the write request from the host 1100 to the secondary volume is denied.

3.3.1.3. Initial-Copying Status

The Initial-Copying status is an intermediate status while the statuschanges from the Simplex status to the Duplex status, and initializationcopy from the primary volume to the secondary volume (copy of datapreviously stored in the primary volume to the secondary volume) isperformed during this period as needed. When the initialization copy iscomplete and the necessary processing for changing to the Duplex statusis complete, the pair status becomes Duplex.

3.3.1.4. Suspend Status

The Suspend status is a status where the contents written into theprimary volume are not reflected in the secondary volume. In thisstatus, the contents of the primary volume data and the contents of thesecondary volume data configuring a pair are not the same. With acommand from the user or the host 1100 as the trigger, the pair statuschanges from another status to the Suspend status. In addition, whensynchronous remote copy can no longer be performed due to a networkfailure or the like between the virtual storage apparatuses 1000, thepair status may also be automatically changed to the Suspend status.

In the ensuing explanation, in the latter case; that is, the Suspendstatus caused by the occurrence of a failure is referred to as thefailure Suspend status. As a representative cause for the pair status tobecome a failure Suspend status, in addition to a network failure, afailure in the primary volume or the secondary volume, or a failure inthe controller 1010 can also be considered.

When the pair status becomes a Suspend status and there is a writerequest to the primary volume after the point in time that the pairstatus become the Suspend status, the copy source storage 1000 receiveswrite data according to the write request and stores this in the primaryvolume, but does not send the write data to the copy destination virtualstorage apparatus 1000. In addition, the copy source virtual storageapparatus 1000 stores the location in the primary volume where the writedata was written as a differential bitmap or the like.

3.3.1.5. Duplex-Pending Status

The Duplex-Pending status is an intermediate status while the statuschanges from the Suspend status to the Duplex status. In this status, inorder to make the contents of the primary volume data and the contentsof the secondary volume data coincide, data is copied from the primaryvolume to the secondary volume. After the contents of the primary volumedata and the contents of the secondary volume data become the same, thepair status becomes Duplex.

The copying of data in the Duplex-Pending status is executed based ondifferential copy of copying only the portions that need to be updated(that is, the non-coinciding portions in the primary volume data and thesecondary volume data) by using the write location (for instance, theforegoing differential bitmap or the like) recorded in the copy sourcevirtual storage apparatus 1000 or the copy destination virtual storageapparatus 1000 during the Suspend status.

Although the Initial-Copying status and the Duplex-Pending status wereexplained to be separate statuses above, these may also be combined intoone status and displayed on the screen of the management host 1200 orchanged to a different status.

4. Example Programs to be Executed with Storage Apparatus 1500 andExample Information to be Managed by Programs

FIG. 3 shows example programs to be executed by the storage apparatus1500 and example information to be managed by the programs, and theseprograms and information perform the same operations as the virtualstorage apparatus 1000.

5. Example Characteristic Configuration of Present Embodiment

In addition to the foregoing configuration, in this example embodiment,the virtual storage apparatuses 1000L, 1000R of the information system100 and the failure detection storage apparatus 1700 are mutuallyconnected via a prescribed network (FIG. 1). The hardware configurationof the failure detection storage apparatus 1700 may be configured fromthe same type of components as the virtual storage apparatus 1000.

The failure detection storage apparatus 1700 provides a part or all ofthe areas of a parity group that uses its own HDD 1030 as the systemcomponent as a failure detection volume 1800 to the virtual storageapparatuses 1000L, 1000R. Here, with the virtual storage apparatuses1000L, 1000R, the same failure detection volume 1800 will have beenprovided as the failure detection virtual volume 3000C(cannot find thisreference number in the FIGS; see other occurrences of this referencenumber in spec) (FIG. 2).

The failure detection storage apparatus 1700 stores failure informationshowing whether a failure has occurred in the system component of theinformation system 100 in the failure detection volume 1800.

Moreover, the programs and information to be executed in the failuredetection storage apparatus 1700 perform the same operations as thevirtual storage apparatus 1000 to the failure detection flag describedlater (FIG. 3).

Specifically, the I/O processing program 6220 of the failure detectionstorage apparatus 1700 performs processing for checking the failuredetection flag (described later) and writing (storing) the failuredetection flag in the corresponding failure detection volume 1800 basedon the control of the processor 1011 of the failure detection storageapparatus 1700 and according to a command from the virtual storageapparatuses 1000L, 1000R (FIG. 3).

Meanwhile, the virtual storage apparatuses 1000L, 1000R retain thefollowing four pieces of information in the copy pair information 6040in order to associate and set the copy pair of the primary volume andsecondary volume of remote copy with the failure detection volume 1800upon creating such pair of the primary volume and secondary volume ofremote copy. Specifically, the copy pair information 6040 of thisembodiment has the following example information, and such informationare associated and set.

(1) Identifier of the virtual storage apparatus 1000 retaining theprimary volume and identifier of the volume

(2) Identifier of the virtual storage apparatus 1000 retaining thesecondary volume and identifier of the volume

(3) Copy pair status

(4) Identifier of the failure detection volume (failure detectionvirtual volume 3000C) and address of data storing the failureinformation

The characteristic processing contents of the I/O processing program6020 of the virtual storage apparatuses 1000L, 1000R in this embodimentwill be described later.

The I/O request processing, which is characteristic processing of remotecopy in this embodiment, is now explained. With the I/O requestprocessing of remote copy in this embodiment, since the I/O is processedwhile avoiding a failure, it is necessary to perform processing whilesatisfying the following requirements.

With the information system 100 in this embodiment, foremost, even if afailure (single point of failure) occurs in one of the system componentsof the information system 100, it is necessary to continue the I/Oprocessing by re-issuing an I/O request from the host 1100 and avoidreplying an I/O error to the host 1100.

In other words, in order to make the availability of the informationsystem 100 in this embodiment higher than a stand-alone virtual storageapparatus, it is necessary to continue the I/O processing in the host1100 even if a single point of failure occurs. With the informationsystem 100, for example, if the I/O request itself is replied as an I/Oerror to the host 1100 due to a remote copy error, the I/O processing inthe host 1100 cannot be continued due to the single point of failure ofthe virtual storage apparatus 1000L. Thus, with the information system100, it is necessary to avoid this kind of processing.

Furthermore, with the information system 100 in this embodiment,secondly, even when a failure (multiple failure) occurs in a pluralityof system components of the information system 100, it is necessary toavoid erroneously accessing old data. Here, the continuity of the I/Oprocessing in the host 1100 is no object.

Here, old data refers to data that is at variance with the data in thevolume 3100 recognized by the host 1100, and a typical example would bethe data in the volume 3000RA of the virtual storage apparatus 1000R ina case where the virtual storage apparatus 1000L (high priority virtualstorage apparatus 1000) is being operated independently after the linkbetween the virtual storage apparatuses 1000L, 1000R is disconnected.

In other words, since the information system 100 of this embodiment is aredundant system of a production system and a backup system, when afailure occurs in a plurality of system components, depending on thefailed portion, there may be cases where the continuation of the I/Oprocessing of the host 1100 becomes impossible.

Here, with the information system 100, when there is time during theoccurrence of the single point of failure to the occurrence of the nextfailure, there may be cases where data (old data) to which the writerequest issued after the occurrence of the single point of failure isnot reflected exists in either the virtual storage apparatus 1000L orthe virtual storage apparatus 1000R.

Accordingly, with the information system 100, it is necessary to preventthe host 1100 from erroneously accessing the old data. With theinformation system 100, for instance, if the remote copy ends in anerror (occurrence of single point of failure) due to the occurrence of asingle point of failure caused by a connection failure or the like dueto a disconnection between the virtual storage apparatuses 1000L, 1000R,considered may be a case of simply processing the I/O requestindependently with the virtual storage apparatus 1000L. Here, with theinformation system 100, when a failure (multiple failure) subsequentlyoccurs in the virtual storage apparatus 1000L a while later, since thehost 1100 reissues the I/O request to the virtual storage apparatus1000R with an I/O error to the virtual storage apparatus 1000L as thetrigger, the host 1100 will access the old data of the virtual storageapparatus 1000R.

Thus, with the information system 100 of this embodiment, a failuredetection storage apparatus 1700 capable of processing a SCSI standardReserve command is connected to the virtual storage apparatuses 1000L,1000R, and comprises a failure detection volume 1800.

If the virtual storage apparatus 1000L detects a failure, it writes afailure information flag showing the detection of a failure in a subblock of the failure detection volume 1800 (failure detection virtualvolume 3000C) that was set during the creation of a pair of the primaryvolume and secondary volume of remote copy, and thereafter continues theI/O processing to the host 1100.

Specifically, when the virtual storage apparatus 1000L detects a remotecopy error, it reserves the failure detection volume 1800 with theReserve command (exclusively secures the failure detection volume 1800so that the other apparatuses will not be able to access the failuredetection volume 1800) before sending an I/O reply to the host 1100.

Subsequently, the virtual storage apparatus 1000L reads a sub block ofthe failure detection volume 1800 that was set during the creation of apair of the primary volume and secondary volume of remote copy, checkswhether a failure information flag is written and writes a failureinformation flag in that sub block if a failure information flag is notwritten.

Subsequently, the virtual storage apparatus 1000L cancels thereservation of the failure detection volume 1800 with a Reservecancellation command, changes to the failure Suspend status and stopsthe remote copy, and resumes the I/O processing to the host 1100.Meanwhile, if the virtual storage apparatus 1000L is not able to reservethe failure detection volume 1800 for a given period of time, or afailure information flag has already been written by the other virtualstorage apparatus 1000R, it replies an I/O error to the host 1100.

Meanwhile, if an I/O request is issued from the host 1100, the virtualstorage apparatus 1000R checks the failure detection volume 1800(failure detection virtual volume 3000C) of the failure detectionstorage apparatus 1700 before starting the I/O processing to the host1100, and starts the I/O processing only when a failure information flaghas not been written.

Specifically, when an I/O request is issued from the host 1100 and thevirtual storage apparatus 1000R receives such I/O request, the virtualstorage apparatus 1000R reserves the failure detection volume 1800 witha Reserve command before sending an I/O reply to the host 1100.

Subsequently, the virtual storage apparatus 1000R reads the sub block ofthe failure detection volume 1800 that was set during the creation of apair of the primary volume and secondary volume of remote copy, checkswhether a failure information flag is written, and writes a failureinformation flag in the sub block when a failure flag is not written.

Subsequently, the virtual storage apparatus 1000R cancels thereservation of the failure detection volume 1800 with a Reservecancellation command, changes to the failure Suspend status and stopsthe remote copy, and resumes the I/O processing to the host 1100 withthe secondary volume. Meanwhile, if the virtual storage apparatus 1000Ris not able to reserve the failure detection volume 1800 for a givenperiod of time, or a failure information flag has already been writtenby the other virtual storage apparatus 1000L, it does not change to thefailure Suspend status and replies an I/O error to the host 1100.

6. Example I/O Request Processing of Virtual Storage Apparatus 1000

FIG. 5 and FIG. 6 are diagrams showing an example operation of remotecopy to be executed by the I/O processing program 6020 when the virtualstorage apparatus 1000 receives an I/O request.

(SP1) The virtual storage apparatus 1000 receives an I/O request.

(SP2) The virtual storage apparatus 1000 determines whether the volumesvolume 3000A, 3000B subject to the I/O request are in a remote copyrelationship, executes SP3 if they are unrelated, and executes SP4 ifthey are related.

(SP3) The virtual storage apparatus 1000 performs the corresponding I/Oprocessing, returns an I/O success reply or I/O error reply to the host1100, and then ends the processing.

(SP4) The virtual storage apparatus 1000 determines the attribute of thevolumes 3000A, 3000B of remote copy subject to the I/O request, executesSP5 if it is a primary volume attribute and executes SP6 if it is asecondary volume attribute. Here, the virtual storage apparatus 1000Lwith the primary volume is the high priority virtual storage apparatus1000. Moreover, if an I/O request is issued to the virtual storageapparatus 1000R with the secondary volume, this is a case where afailure has occurred in the virtual storage apparatus 1000L with theprimary volume and, in this case, the remote copy is not operating.

(SP5) The virtual storage apparatus 1000 determines the contents of theI/O subject to the I/O request, and executes SP3 if it is a read requestand executes SP6 if it is a write request.

(SP6) The virtual storage apparatus 1000 executes the synchronous remotecopy processing, transfers write data to the virtual storage apparatus1000R, and waits for a reply.

(SP7) The virtual storage apparatus 1000 determines whether thesynchronous remote copy processing was successful, and executes SP11 ifit is successful and executes SP8 if it is an error.

(SP8) The virtual storage apparatus 1000 changes the copy pair status ofthe remote copy in which the target volume is a primary volume to thefailure Suspend status, and stops the remote copy. In this case, thevirtual storage apparatus 1000 does not prohibit the writing of datainto that volume.

(SP9) The virtual storage apparatus 1000 writes a failure informationflag in the sub block of the failure detection volume 1800 that was setduring the creation of a pair of the primary volume and secondary volumeof remote copy.

(SP10) The virtual storage apparatus 1000 determines whether the writingof the failure information flag was successful, and executes SP11 if itwas successful and executes SP12 is it was an error. If a failureinformation flag has already been written in the corresponding failuredetection volume 1800, since this means that the other virtual storageapparatus 1000 has written the failure information flag, the writingwill be an error. Meanwhile, if the other virtual storage apparatus 1000has not written the failure information flag, the failure informationflag is written and the writing will be successful.

(SP11) The virtual storage apparatus 1000 performs the I/O processing(write processing), returns an I/O success reply to the host 1100, andthen ends the processing.

(SP12) The virtual storage apparatus 1000 returns an I/O error reply tothe host 1100, and then ends the processing. Subsequently, if the host1100 the retries the foregoing process but still results in an I/Oerror, it issues an I/O request to the virtual storage apparatus 1000Ras the low priority virtual storage apparatus 1000.

(SP13) The virtual storage apparatus 1000 changes the copy pair statusof the remote copy in which the target volume is a secondary volume tothe failure Suspend status, and stops the remote copy. In this case, thevirtual storage apparatus 1000 does not prohibit the writing of datainto that volume.

(SP14) The virtual storage apparatus 1000 writes a failure informationflag in the sub block of the failure detection volume 1800 that was setduring the creation of a pair of the primary volume and secondary volumeof remote copy.

(SP15) The virtual storage apparatus 1000 determines whether the writingof the failure information flag was successful, and executes SP11 if itwas successful and executes SP12 is it was an error. If a failureinformation flag has already been written in the corresponding failuredetection volume 1800, since this means that the other virtual storageapparatus 1000 has written the failure information flag, the writingwill be an error. Meanwhile, if the other virtual storage apparatus 1000has not written the failure information flag, the failure informationflag is written and the writing will be successful.

Like this, with the information system 100, the virtual storageapparatuses 1000L, 1000R respectively set a copy pair of the primaryvolume and secondary volume in the copy pair information 6040 accordingto an external command from the management host 1200 or the like,associates the failure detection volume 1800 with that copy pair, sendsan I/O request directed to the secondary volume to the virtual storageapparatus 1000R if the I/O request to the primary error ends in an errorin the host 1100, and stores the failure information flag in the failuredetection volume 1800 if the virtual storage apparatuses 1000L, 1000Rdetect a failure in the virtual storage apparatuses 1000R, 1000L of theother side or a connection failure between the virtual storageapparatuses 1000L, 1000R.

Accordingly, since an I/O request is sent to the second volume when anI/O request to the first volume ends in an error, it is possible toeffectively prevent the execution of processing to an I/O request of thehost 1100 based on data of the second volume, which is not up to date,as a result of the remote copy with the first volume not being performedand data of the second volume not being up to date.

Moreover, with the information system 100, when the virtual storageapparatus 1000R receives an I/O request from the host 1100, itdetermines whether a failure information flag has been written into thefailure detection volume 1800 associated with the copy paircorresponding to that I/O request, and sends an I/O request error replyif the failure information flag is written.

Accordingly, since an I/O request is sent to the second volume when anI/O request to the first volume ends in an error, it is possible toeffectively prevent the execution of processing to an I/O request of thehost 1100 based on data of the second volume, which is not up to date,as a result of the remote copy with the first volume not being performedand data of the second volume not being up to date.

Further, with the information system 100, if the virtual storageapparatus 1000L detects an error in the remote copy to the secondaryvolume, it determines whether a failure information flag is written inthe failure detection volume 1800, writes the failure information flagin the failure detection volume 1800 if the failure information flag isnot written, and thereafter executes processing corresponding to the I/Orequest.

Accordingly, even if a failure occurs in the virtual storage apparatus1000L, it is possible to notify the virtual storage apparatus 1000R thatthe remote copy with the primary volume has not been performed and thesecondary volume data is not up to date.

In addition, with the information system 100, if the failure informationflag is written in the virtual storage apparatus 1000L, it sends an I/Orequest error reply.

Accordingly, if the failure information flag is written in the failuredetection volume 1800, since this shows that the independent operationbased on the secondary volume has already been started due to some kindof factor, and that the primary volume data is not up to date, it ispossible to prevent the host 1100 from executing processing to the I/Orequest based on the primary volume data that is not up to date.

Although this example embodiment explained a case of providing the samefailure detection volume 1800 as the failure detection virtual volume3000C to the virtual storage apparatuses 1000L, 1000R, and accessing thefailure detection volume 1800, the present invention is not limited tothe foregoing configuration, and, without providing the failuredetection virtual volume 3000C, the virtual storage apparatuses 1000L,1000R may also directly access the failure detection volume 1800 of thefailure detection storage apparatus 1700, and various other methods mayalso be adopted.

1. An information system, comprising: a host computer; a first storageapparatus connected to the host computer and including a first volume; asecond storage apparatus connected to the host computer and the firststorage apparatus and including a second volume; and a third storageapparatus connected to the first storage apparatus and the secondstorage apparatus and including a third volume; wherein the first andthe second storage apparatuses respectively and internally set a remotecopy pair for copying data of the first volume to the second volume andassociate the third volume with the remote copy pair according to anexternal command; wherein, when an I/O request to the first volume endsin an error, the host computer sends an I/O request directed to thesecond volume to the second storage apparatus; and wherein, if the firstand the second storage apparatuses detect a failure in the first or thesecond storage apparatus of the other side or a connection failurebetween the first and the second storage apparatuses, the first and thesecond storage apparatuses store in the third volume a failureinformation flag showing that the failure was detected.
 2. Theinformation system according to claim 1, wherein, upon receiving the I/Orequest from the host computer, the second storage apparatus determineswhether the failure information flag is stored in the third volumeassociated with the remote copy pair corresponding to the I/O request,and sends an error reply of the I/O request to the host computer if thefailure information flag is stored in the third volume.
 3. Theinformation system according to claim 2, wherein, upon receiving the I/Orequest from the host computer, the second storage apparatus stores thefailure information flag in the third volume if the failure informationflag is not stored in the third volume, and thereafter executesprocessing corresponding to the I/O request.
 4. The information systemaccording to claim 2, wherein, upon detecting an error in the remotecopy to the second volume, the first storage apparatus determineswhether the failure information flag is stored in the third volume,stores the failure information flag in the third volume if the failureinformation flag is not stored in the third volume, and thereafterexecutes processing corresponding to the I/O request.
 5. The informationsystem according to claim 4, wherein, upon receiving the I/O requestfrom the host computer, the first storage apparatus sends an error replyof the I/O request to the host computer if the failure information flagis stored in the third volume.
 6. The information system according toclaim 2, wherein the first and the second storage apparatusesrespectively comprise a plurality of storage devices, a controller, anda cache memory; and wherein the first and the second volumes arerespectively configured from a partial area of the plurality of storagedevices.
 7. The information system according to claim 6, furthercomprising: a fourth storage apparatus connected to the first storageapparatus, configured from a plurality of storage devices, a controllerand a cache memory, and including a fourth volume configured from apartial area of the plurality of storage devices; and a fifth storageapparatus connected to the second storage apparatus, configured from aplurality of storage devices, a controller and a cache memory, andincluding a fifth volume configured from a partial area of the pluralityof storage devices; wherein the first storage apparatus defines a firstvirtual volume as a virtual volume in which its substance exists in thefourth volume of the fourth storage apparatus; wherein the secondstorage apparatus defines a second virtual volume as a virtual volume inwhich its substance exists in the fifth volume of the fifth storageapparatus; wherein the first and the second storage apparatusesrespectively and internally set a remote copy pair for copying data ofthe first virtual volume to the second virtual volume and associate thethird volume with the remote copy pair; and wherein, when an I/O requestto the first virtual volume ends in an error, the host computer sends anI/O request directed to the second virtual volume to the second storageapparatus.
 8. The information system according to claim 2, wherein thefirst and the second storage apparatuses define a third virtual volumeas a virtual volume in which its substance exists in the third volume ofthe third storage apparatus; and wherein, upon receiving the I/O requestfrom the host computer, the second storage apparatus determines whetherthe failure information flag is stored in the third virtual volumeassociated with the remote copy pair corresponding to the I/O request,and sends an error reply of the I/O request to the host computer if thefailure information flag is stored in the third virtual volume.
 9. AnI/O processing method in an information system, comprising: a hostcomputer; a first storage apparatus connected to the host computer andincluding a first volume; a second storage apparatus connected to thehost computer and the first storage apparatus and including a secondvolume; and a third storage apparatus connected to the first storageapparatus and the second storage apparatus and including a third volume;wherein the I/O processing method comprises: a first step of the firstand the second storage apparatuses respectively and internally setting aremote copy pair for copying data of the first volume to the secondvolume and associating the third volume with the remote copy pairaccording to an external command; a second step of the host computersending, when an I/O request to the first volume ends in an error, anI/O request directed to the second volume to the second storageapparatus; and a third step of the first and the second storageapparatus storing in the third volume, if a failure in the first or thesecond storage apparatus of the other side or a connection failurebetween the first and the second storage apparatuses is detected, afailure information flag showing that the failure was detected.
 10. TheI/O processing method of an information system according to claim 9,further comprising: a fourth step of the second storage apparatusdetermining, upon receiving the I/O request from the host computer,whether the failure information flag is stored in the third volumeassociated with the remote copy pair corresponding to the I/O request,and sending an error reply of the I/O request to the host computer ifthe failure information flag is stored in the third volume.
 11. The I/Oprocessing method of an information system according to claim 10,wherein, at the fourth step, upon receiving the I/O request from thehost computer, the second storage apparatus stores the failureinformation flag in the third volume if the failure information flag isnot stored in the third volume, and thereafter executes processingcorresponding to the I/O request.
 12. The I/O processing method of aninformation system according to claim 10, further comprising: a fifthstep of the first storage apparatus determining, upon detecting an errorin the remote copy to the second volume, whether the failure informationflag is stored in the third volume, storing the failure information flagin the third volume if the failure information flag is not stored in thethird volume, and thereafter executing processing corresponding to theI/O request.
 13. The I/O processing method of an information systemaccording to claim 12, wherein, at the fifth step, upon receiving theI/O request from the host computer, the first storage apparatus sends anerror reply of the I/O request to the host computer if the failureinformation flag is stored in the third volume.
 14. The I/O processingmethod of an information system according to claim 10, wherein the firstand the second storage apparatuses respectively comprise a plurality ofstorage devices, a controller, and a cache memory; and wherein the firstand the second volumes are respectively configured from a partial areaof the plurality of storage devices.
 15. The I/O processing method of aninformation system according to claim 10, wherein the information systemfurther comprises: a fourth storage apparatus connected to the firststorage apparatus, configured from a plurality of storage devices, acontroller and a cache memory, and including a fourth volume configuredfrom a partial area of the plurality of storage devices; and a fifthstorage apparatus connected to the second storage apparatus, configuredfrom a plurality of storage devices, a controller and a cache memory,and including a fifth volume configured from a partial area of theplurality of storage devices; wherein, at the first step, the firststorage apparatus defines a first virtual volume as a virtual volume inwhich its substance exists in the fourth volume of the fourth storageapparatus, the second storage apparatus defines a second virtual volumeas a virtual volume in which its substance exists in the fifth volume ofthe fifth storage apparatus, and the first and the second storageapparatuses respectively and internally set a remote copy pair forcopying data of the first virtual volume to the second virtual volumeand associate the third volume with the remote copy pair; and wherein,at the second step, when an I/O request to the first virtual volume endsin an error, the host computer sends an I/O request directed to thesecond virtual volume to the second storage apparatus.
 16. The I/Oprocessing method of an information system according to claim 10,wherein, at the first step, the first and the second storage apparatusesdefine a third virtual volume as a virtual volume in which its substanceexists in the third volume of the third storage apparatus; and wherein,at the fourth step, upon receiving the I/O request from the hostcomputer, the second storage apparatus determines whether the failureinformation flag is stored in the third virtual volume associated withthe remote copy pair corresponding to the I/O request, and sends anerror reply of the I/O request to the host computer if the failureinformation flag is stored in the third virtual volume.